1. Field of the Invention
The present invention is directed to the general field of video processing and to the more specific field of processing of segmented video. In particular, the invention is concerned with the maintenance of background models in segmented video and classifying changes to the background model.
2. Related Art
Many video processing applications require segmentation of video objects (i.e., the differentiation of legitimately moving objects from the static background scene depicted in a video sequence). Such applications include, for example, video mosaic building, object-based video compression, object-based video editing, and automated video surveillance. Many video object segmentation algorithms use video scene background models (which can simply be referred to as “background models”) as an aid. The general idea is that each frame of a video sequence can be registered to the background model and compared, pixel-by-pixel, to the background model. Pixels that display sufficient difference are considered foreground, or moving, pixels. However, there are a Wide range of phenomena that can cause pixel-level changes, such as: unstable backgrounds (e.g., rippling water, blowing leaves, etc.); lighting phenomena (e.g., clouds moving across the sun, shadows, etc.); and camera phenomena (e.g., automatic gain control (AGC), auto iris, auto focus, etc.).
Using video object segmentation (or a variation thereof), objects, or parts of objects, that exhibit independent motion can usually be detected. There are two basic problems that arise when objects in a scene are stationary for a long period of time, and either of these two phenomena can degrade the performance of video object segmentation for any application.
First, if an object remains stationary for a long period of time, the object could be “permanently” detected as a foreground object. However, for all practical purposes, the object has become part of the background. In FIG. 1A, this problem is illustrated for a car 11 that drives into the video sequence and parks therein. The car is continually monitored as a foreground object 12 but has actually become part of the background (i.e., “permanent” segmentation).
Second, if an object, initially stationary, is part of the background model (e.g., gets “burned in”) and then moves, the object exposes a region of the background model (e.g., static background) that has not been modeled. The exposed region of the background model is erroneously detected as a foreground object. In FIG. 1B, this problem is illustrated for a parked car 13 that drives out of the video sequence. The car 13 exposes a car-shaped “hole” 14 segmented in the background model.
As discussed, for example, in U.S. patent application Ser. No. 09/472,162, titled “Method, Apparatus, and System for Compressing/Decompressing Digital Video Data,” filed Dec. 27, 1999, and U.S. patent application Ser. No. 09/609,919, titled “Scene Model Generation from Video for Use in Video Processing,” filed Jul. 3, 2000 (both commonly assigned, and incorporated herein by reference), when building photo mosaics, video mosaics, or video scene models, it is often desirable to extract those portions of the source images that represent “true” background. For example, a parked car in a video sequence (or any other collection of images) that remains parked for the duration of the video sequence may be considered true background. However, a car in a video sequence that is initially parked and later drives away at some point in the video sequence should properly be considered “not background.”
If care is not taken to identify true background regions, artifacts will result. If the goal is to produce a mosaic or background image, foreground objects can be “burned in” the background model resulting in unnatural-looking imagery. If the goal is to build a scene model as a basis for video segmentation, the results can be poor segmentations, where parts of foreground objects are not detected, and where some exposed background regions are detected as foreground objects.
FIG. 2 illustrates a prior art example of allowing foreground objects to corrupt a background model. The video sequence depicts a golfer preparing to tee off. A subset 21 of the source images from the video sequence depict a part of this video sequence. The source images are used to generate a background model 22 and foreground objects 23. However, the background model 22 contains foreground objects 23 (e.g., the golfer on the left, and part of the golfer's shirt on the right) burned into the background model 22, and the foreground objects 23 are incompletely segmented (e.g., part of the golfer's torso, and part of the golf club).